Memory system and method

ABSTRACT

According to an embodiment, a memory system includes a controller which includes an interface connectable with a host with cache coherency kept. The controller is configured to: before the host writes a command to an I/O submission queue, read the I/O submission queue; after the reading, detect via the interface an invalidation request, the invalidation request being based on writing of the command by the host to the I/O submission queue; and in response to the invalidation request, acquire the command in the I/O submission queue.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2020-156861, filed on Sep. 18, 2020; theentire contents of which are incorporated herein by reference.

FIELD

An embodiment described herein relates generally to a memory system anda method.

BACKGROUND

In a system which includes a host and a storage device, when the hostregisters a command in an SQ region which is a submission queue, inresponse to this, the storage device acquires the command. In this case,it is desired that the command is acquired with low latency.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a configuration of an informationprocessing system which includes a memory system according to anembodiment;

FIG. 2 is a diagram illustrating one example of a configuration of ahost memory according to the embodiment;

FIG. 3 is a block diagram illustrating one example of a functionconfiguration of a controller according to the embodiment;

FIG. 4 is a sequence diagram illustrating operation of an informationprocessing system which includes the memory system according to theembodiment; and

FIG. 5 is a sequence diagram explaining a procedure of commandacquisition processing in a comparative example.

DETAILED DESCRIPTION

According to an embodiment, a memory system includes a controller whichincludes an interface connectable with a host with cache coherency kept.The controller is configured to: before the host writes a command to anI/O submission queue, read the I/O submission queue; after the reading,detect via the interface an invalidation request, the invalidationrequest being based on writing of the command by the host to the I/Osubmission queue; and in response to the invalidation request, acquirethe command in the I/O submission queue.

Hereinafter, a memory system and a method according to an embodimentwill be described in detail with reference to the accompanying drawings.Note that the present invention is not limited by the embodiment.

Embodiment

FIG. 1 is a diagram illustrating a configuration of an informationprocessing system 1 which includes a memory system 100 according to theembodiment.

The memory system 100 is communicably connected to a host 2 andfunctions as an external storage device for the host 2. The memorysystem 100 is a flash memory for embedded use in conformity with aUniversal Flash Storage (UFS) standard or an embedded Multi Media Card(eMMC) standard, a Solid State Drive (SSD), or the like. The host 2 is,for example, a microprocessor, an imaging device, or the like.Hereinafter, a case in which the host 2 and the memory system 100 cantransmit and receive a packet in conformity with a Non-Volatile Memoryexpress (NVMe) (registered trademark) standard via a communication pathwill be illustrated as an example.

The host 2 has a host System on Chip (SoC) 3 and a host memory 4. Thehost SoC3 has a CPU and performs processing required to issue a requestto the memory system 100. For example, the host SoC3 writes a commandfor creasing a queue and a command for data updating or the like to thehost memory 4 and performs reading and writing data in a controlregister of the memory system 100.

In addition, the host SoC3 has an interface in conformity with aprotocol, such as a Compute Express Link (CXL) (registered trademark)protocol, which guarantees cache coherency. The host SoC3 monitorsaccess to the host memory 4 from the host SoC3 itself and the memorysystem 100 for tracking memory states. In accordance with the states,the host SoC3 issues a request to keep the cache coherency to the memorysystem 100. For example, when the host SoC3 performs writing to a regionof the host memory 4, which is read by using a request (a Rd Sharerequest of CXL.Cache) which indicates that the memory system 100performs reading share, the host SoC3 transmits an invalidation request(a Snvinv request of CXL. Cache) of corresponding data to the memorysystem 100 by a function of the above-mentioned interface (for example,a bridge function of the CXL standard).

The host memory 4 is, for example, a memory such as a Dynamic RandomAccess Memory (DRAM). Here, with reference to FIG. 2, one example of aconfiguration of the host memory 4 will be described. FIG. 2 is adiagram illustrating one example of the configuration of the host memory4 according to the embodiment. As illustrated in FIG. 2, the host memory4 includes Admin SQ 41, Admin CQ 42, I/O SQ 43, I/O CQ 44, and a bufferregion 45. The Admin SQ 41 is a management command submission queue. TheAdmin CQ 42 is a management command completion queue. The I/O SQ 43 isan I/O command submission queue and a submission queue as to datatransfer. The I/O CQ 44 is a command completion queue as to the datatransfer. The buffer region 45 has stored therein write datacorresponding to a write command enqueued to the I/O SQ 43 and read datacorresponding to a read command.

As described above, each of the queue of the management command and theI/O command queue is configured with a pair of the submission queue andthe completion queue. The submission queue is a queue which has storedtherein a command submitted from the host 2 to the memory system 100.The completion queue is a queue in which when the memory system 100completes execution of a command received via the submission queue,notification of execution completion of the above-mentioned command isstored by the memory system 100. The host 2 can learn the completion ofthe execution of the command by confirming the notification stored inthe completion queue.

Each of the submission queue and the completion queue is a ring bufferas one example. An entry which each of the queues (each of thesubmission queue and the completion queue) includes is managed by twovariables which are Head and Tail. Based on locations which Head andTail indicate (a head pointer and a tail pointer), the host 2 or thememory system 100 can identify a reference location of each of thequeues (for example, a location of newly enqueued command information).

With reference back to FIG. 1, the memory system 100 has a controller110, a buffer memory 120, and a nonvolatile memory 130. The controller110 has a host interface 111, a buffer interface 112, a CPU 113, aRandom Access Memory (RAM) 114, and a memory interface 115.

The nonvolatile memory 130 can include a plurality of memory chips. Eachof the memory chips has a memory cell array. In the memory cell array ofeach of the memory chips, a unit with which access for collectivewriting and reading of data is enabled is a physical page. A physicalblock includes a plurality of physical pages and is a minimum accessunit with which data can be independently erased. It is supposed thatdata in the nonvolatile memory 130 is managed by the CPU 113 by using acluster which is a data management unit smaller than a unit of onephysical page. A cluster size is equal to or greater than a size of asector which is a minimum access unit from the host 2 and it is definedthat a natural number multiple of the cluster size is a physical pagesize. One physical page can include four clusters and one logical pagecan include 64 clusters.

For example, a size of the above-mentioned writing unit may be a clustersize, a page size, a block size, and a size different from the clustersize, the page size, and the block size. The cluster size may be 4 KBand the page size may be 16 KB.

The nonvolatile memory 130 is, for example, a NAND type flash memory andmay be other nonvolatile semiconductor memory such as a NOR type flashmemory, a Magnet Resistive Random Access Memory (MRAM), a Phase-changeRandom Access Memory (PRAM), a Resistive Random Access Memory (ReRAM),and a Ferroelectric Random Access Memory (FeRAM). In addition, thenonvolatile memory 130 may also be a memory having a three-dimensionalstructure.

The host SoC3 and the host interface 111 operate in accordance with aprotocol specified by the NVMe standard. In addition, each of the hostSoC3 and the host interface 111 also has an interface in conformity witha protocol, such as a CXL protocol, which guarantees the cachecoherency. Note that besides the CXL protocol, the protocol which theabove-mentioned interface is in conformity with may be a Cache CoherentInterconnect for Accelerators (CCIX) protocol or an Open CoherentAccelerator Processor Interface (OpenCAPI) (a registered trademark)protocol. Note that in a case of the CXL protocol, it is required tosupport CXL.cache in addition to CXL.io.

The CPU 113 controls each of units included in the controller 110 in anoverall manner. The CPU 113 acquires a variety of commands, for example,via the host interface 111 and upon acquiring each of the variety ofcommands, performs processing in accordance with the acquired command.

The RAM 114 is a memory and functions as a working area by the CPU 113.

The buffer memory 120 functions as a memory area which temporarily holdsinformation. The buffer memory 120 is, for example, a general-purposememory such as a Static Random Access Memory (SRAM) and a Dynamic RandomAccess Memory (DRAM). In addition, the buffer memory 120 may beinstalled inside the controller 110 or may be installed outside thecontroller 110, independently from the controller 110.

The memory interface 115 is connected to the nonvolatile memory 130 viaa bus and controls communication with the nonvolatile memory 130. Forexample, when instructed to execute a write command from the CPU 113,the memory interface 115 transfers a write request and write data to thenonvolatile memory 130. In addition, upon completing write processing,the nonvolatile memory 130 transmits completion notification to thememory interface 115, and the memory interface 115 receives thecompletion notification from the nonvolatile memory 130. When instructedto execute a read command from the CPU 113, the memory interface 115transfers a read request to the nonvolatile memory 130, and thenonvolatile memory 130 transmits read data to the memory interface 115,and the memory interface 115 receives the read data from the nonvolatilememory 130.

When having received, via the memory interface 115, the completionnotification for the write request instructed by the CPU 113, the CPU113 registers completion information indicating execution completion ofthe write command via the host interface 111 in the I/O CQ 44 in thehost memory 4. In a case of a read request, the CPU 113 transfers readdata via the host interface 111 to the buffer region in the host memory4 and registers completion information indicating execution completionof a read command via the host interface 111 in the I/O CQ 44 in thehost memory 4.

The buffer interface 112 is a controller which controls access to thebuffer memory 120 from the controller 110 and is a controller whichenables the access to the buffer memory 120 at, for example, a DoubleData Rate (DDR). The CPU 113 stores, in the buffer memory 120, the writedata transferred to the nonvolatile memory 130 and the read dataoutputted by the nonvolatile memory 130 as needed and performs datatransfer via the host interface 111 between the buffer memory 120 andthe host memory 4.

Subsequently, with reference to FIG. 3, functions of the controller 110will be described. FIG. 3 is a block diagram illustrating one example ofa function configuration of the controller 110 according to theembodiment. As illustrated in FIG. 3, the controller 110 includes aninterface module 11, an initial setting module 12, a first-time readingmodule 13, an invalidation request detection module 14, a commandacquisition module 15, and a command processing module 16. The CPU 113executes a program and the like stored in the nonvolatile memory 130 andrealizes the initial setting module 12, the first-time reading module13, and the command processing module 16 by controlling the hostinterface 111, the buffer interface 112, and the memory interface 115.In addition, the invalidation request detection module 14 and thecommand acquisition module 15 are realized by functions implemented inthe host interface 111.

The interface module 11 is an interface (for example, an interface inconformity with the CXL standard) which connects to the host 2 with thecache coherency kept and is realized by the host interface 111.

The initial setting module 12 is an element which performs settingprocessing as to initialization processing. Here, the initializationprocessing is preprocessing executed in a case where a power source ofthe host 2 is turned on, a case where a power source of the memorysystem 100 is turned on, a case where the memory system 100 is restoredfrom a sleep state, or other case.

The sleep state of the memory system 100 is, for example, a state inwhich a power source of at least one part of circuitry and memoriesincluded in the memory system 100 is turned off in order to save powerconsumption of the memory system 100. As a method of realizing the sleepstate of the memory system 100, for example, there are a method in whichthe controller 110 stores predetermined information stored in the RAM114 and the buffer memory 120 into the nonvolatile memory 130 andthereafter, a power source of the memory system 100 is turned off and amethod in which the controller 110 stores predetermined informationstored in the RAM 114 and the buffer memory 120 into a memory circuit,among memory circuits included in the controller 110, whose powerconsumption is small and thereafter, power sources of the circuits andthe memories included in the memory system 100, other than the memorycircuit whose power consumption is small, are turned off.

As one initialization processing, for example, there is creation of theI/O SQ 43. When detecting that the tail pointer of the Admin SQ 41 isupdated by writing processing performed by the host 2, the initialsetting module 12 recognizes that a new command has been registered onthe Admin SQ 41. The initial setting module 12 acquires information ofthe above-mentioned new command from the host memory 4 and analyzes thenew command. When as a result of analyzing the new command, determiningthat the new command is a creation command of the I/O SQ 43, the initialsetting module 12 performs region setting processing for the I/O SQ 43.In this region setting processing for the I/O SQ 43, an address and asize of a region for the I/O SQ 43 are set to the host interface 111,thereby allowing the subsequent invalidation request detection module 14and command acquisition module 15 to issue a request to the I/O SQ 43 onthe host memory 4.

After the region setting processing for the I/O SQ 43 by the initialsetting module 12, the first-time reading module 13 executes processingof reading the above-mentioned I/O SQ 43. After the region settingprocessing for the I/O SQ 43 and before command writing to the I/O SQ 43by the host 2, the first-time reading module 13 reads the I/O SQ 43. Thefirst-time reading module 13 reads the I/O SQ 43 by using a request (aRdShare request of CXL.Cache), whereby the host SoC3 of the host 2recognizes that the memory system 100 references the I/O SQ 43 andstores the referencing of the I/O SQ 43 made by the memory system 100.When in subsequent processing, writing to the I/O SQ 43 occurs, the hostSoC3 issues an invalidation request of corresponding data to the memorysystem 100. In addition, since read information does not indicate validcommand information, the first-time reading module 13 discards the readinformation at predetermined timing.

After the reading made by the first-time reading module 13, theinvalidation request detection module 14 is an element which detects,via the interface module 11, an invalidation request (a Snvinv requestof CXL.Cache) generated by command writing to the I/O SQ 43 made by thehost SoC3. In response to the above-mentioned invalidation request, theinvalidation request detection module 14 detects writing to the I/O SQ43 and updates the tail pointer of the I/O SQ 43. The invalidationrequest detection module 14 is realized by a circuit inside the hostinterface 111.

The command acquisition module 15 is an element which acquires commandinformation of the I/O SQ 43 in response to the invalidation requestdetected by the invalidation request detection module 14. In order toacquire the command information, as with the first-time reading module13, by performing reading by using a request (a RdShare request ofCXL.Cache) which indicates reading share, the host SoC3 of the host 2recognizes that the memory system 100 references the latest data at thatpoint in time. The command acquisition module 15 is realized by acircuit inside the host interface 111.

The command processing module 16 is an element which processes a requestof the host which the command acquired by the command acquisition module15 indicates. The CPU 113 controls the host interface 111, the bufferinterface 112, and the memory interface 115 as needed and performs datatransfer between the host memory 4 and the nonvolatile memory 130.

Next, with reference to FIG. 4, operation of the information processingsystem 1 which includes the memory system 100 will be described. FIG. 4is a sequence diagram illustrating the operation of the informationprocessing system 1 which includes the memory system 100, according tothe embodiment. First, in the information processing system 1, in stepS1 to step S14, initialization processing is first executed. Note thatoperation illustrated in FIG. 4 is operation which is unique to theinformation processing system 1 and in order to operate the memorysystem 100, operations of initialization processing other than thoseillustrated in FIG. 4, which are not unique to the informationprocessing system 1, are required. These processes are conducted atappropriate timing before and after the initialization processingillustrated in FIG. 4. In addition, although in FIG. 4, as commandprocessing, a read command and a write command are described asexamples, also as to commands other than the read command and the writecommand, command inputting between the host SoC3 and the memory system100 and a procedure of the command completion are similar thereto andthe CPU 113 performs control in accordance with necessity, therebyappropriately processing a request of the host by the command.

The host SoC3 of the host 2 writes a creation command of the I/O SQ 43to the Admin SQ 41 (step S1), executes a memory barrier instruction (forexample, a SFENCE instruction implemented in a CPU manufactured by IntelCorporation) (step S2), and updates a tail pointer of the Admin SQ 41(step S3). The memory barrier instruction is an instruction needed toguarantee order of writing to the Admin SQ 41 and updating of the tailpointer of the Admin SQ 41 (step S3). The host SoC3 updates the tailpointer of the Admin SQ 41, whereby the initial setting module 12recognizes that a new command has been registered (step S4). Inaccordance with the recognition, the initial setting module 12 accessesthe host memory 4 and makes a request to read a command (step S5) andacquires command information from the host memory 4 (step S6). Based onthe acquired command information, the initial setting module 12 performssetting processing for the I/O SQ 43 (step S7). The initial settingmodule 12 can set an address and a size of the I/O SQ 43 by performingthis setting processing.

After the above-mentioned setting processing, the first-time readingmodule 13 accesses the host memory 4 and transmits a request to read thewhole region of the I/O SQ 43 to the host 2 (step S8). The first-timereading module 13 transmits, for example, a request (a RdShare request)multiple times, which is a request for reading data having a 64-byteslength corresponding to a command length and a cache management unit,and which indicates reading share of a CXL.Cache. The first-time readingmodule 13 acquires data of the I/O SQ 43 from the host memory 4 (stepS9). As described above, prior to writing of the command (for example,the write command and the read command) to the I/O SQ 43 made by thehost SoC3, which is executed after finishing the initializationprocessing, the first-time reading module 13 reads the data of the I/OSQ 43. Note that based on the address and the size of the I/O SQ 43,which is set by the initial setting module 12, the first-time readingmodule 13 processes reading.

The first-time reading module 13 discards the acquired data of the I/OSQ 43 (step S10). As described above, the first-time reading module 13dummy reads the I/O SQ 43. The initial setting module 12 writescompletion notification to the Admin CQ 42 of the host memory 4 (stepS11) and notifies the host SoC3 of an interrupt (step S12).

The host SoC3 reads information of the Admin CQ (step S13) and acquirescommand completion information (step S14). The processes up to herecorrespond to the initialization processing.

The host SoC3 writes the command to the I/O SQ 43 (step S15). As aresult of this, updating of the I/O SQ 43 read by the first-time readingmodule 13 occurs, and the host SoC3 issues an invalidation request tothe memory system 100 by a CXL-standard coherency bridge function of thehost SoC3 (step S16). For example, the host SoC3 transmits theinvalidation request (a Snvinv request) of CXL.Cache. In addition, thisinvalidation request includes address information which indicates alocation where the host SoC3 writes the command (for example, the writecommand or the read command) in step S15. When the host SoC3 writes thecommand to the I/O SQ 43, it is required to atomically perform thewriting by one instruction in which pieces of data including the commandinformation are collected, and it is required to use, for example, anAVX-512 store instruction which is implemented in a CPU manufactured byIntel Corporation.

The invalidation request detection module 14 of the memory system 100detects, via the interface module 11, the invalidation request issued bythe host SoC3. In addition, the invalidation request detection module 14references the address information included in the information of theabove-mentioned invalidation request and based on the addressinformation, updates a tail pointer of the I/O SQ 43 (step S17).

Based on the updated tail pointer of the I/O SQ 43, the commandacquisition module 15 issues a command reading request to the hostmemory 4 (step S18) and acquires command information from the hostmemory 4 (step S19). The command acquisition module 15 acquires thecommand information and updates a head pointer of the I/O SQ 43 (stepS20).

In accordance with contents which the acquired command informationindicates, the command processing module 16 executes command processing(step S21). In step S21, when the acquired command information indicatesthe read command, based on the contents of the command information, thecommand processing module 16 reads corresponding data from thenonvolatile memory 130 and transfers the data to the host memory 4.

In addition, when the acquired command information indicates the writecommand, in step S21, the command processing module 16 acquires writedata from the host memory 4 and based on contents of the commandinformation, writes the acquired write data to a corresponding locationof the nonvolatile memory 130.

Upon completing the processing of the command, the command processingmodule 16 writes completion notification to the I/O CQ 44 of the hostmemory 4 (step S22) and notifies the host SoC3 of an interrupt (stepS23).

In response to the notification of the interrupt, the host SoC3 readsthe I/O CQ 44 of the host memory 4 (step S24) and acquires commandcompletion information from the host memory 4 (step S25), finishing theprocessing.

As described above, in the embodiment, the interface module 11 connectsto the host 2 with the cache coherency kept. Prior to the commandwriting to the I/O SQ 43 made by the host 2, the first-time readingmodule 13 reads the I/O SQ 43. After the reading made by the first-timereading module 13, the invalidation request detection module 14 detects,via the interface module 11, the invalidation request that is based onthe command writing to the I/O SQ 43 made by the host 2 and in responseto the invalidation request, the command acquisition module 15 acquiresa command of the I/O SQ 43. As described above, without detectingupdating of the tail pointer of the I/O SQ 43 by the host 2, thecontroller 110 acquires the command information, thus allowing thecommand to be acquired with low latency.

As a technology (referred to as a comparative example) which is comparedwith the technology in the embodiment, there is a technology in which ahost SoC writes a command to an I/O SQ and further, the host updates anI/O SQ tail pointer. Here, with reference to FIG. 5, a procedure ofcommand acquisition processing in the comparative example will bedescribed. FIG. 5 is a sequence diagram explaining a procedure ofcommand acquisition processing in a comparative example.

It is premised that creation of the I/O SQ has been completed. First, ahost SoC writes the command to the I/O SQ (step S51) and executes amemory barrier instruction (step S52). This memory barrier instructionis an instruction which guarantees order of writing to the I/O SQ andupdating of the I/O SQ tail pointer. Subsequently, the host SoC updatesthe I/O SQ tail pointer (step S53).

Based on updating of the I/O SQ tail pointer, a memory system recognizesnew command supply (step S54), issues a request to read the command ofthe I/O SQ to a host memory (step S55), and acquires command information(step S56). The memory system increments a head pointer of the I/O SQ(step S57).

According to the comparative example, a delay up to when the memorysystem starts to read the command occurs due to the time required forupdating the tail pointer of the I/O SQ made by the host SoC (step S53in FIG. 5) and the time required for executing the instruction (step S52in FIG. 5) which guarantees the order of processing.

In contrast to the comparative example, according to the embodiment,since after the invalidation request has been detected, the memorysystem 100 updates the I/O SQ tail pointer, it is not required for thehost SoC3 to update the tail pointer and it is also not required for thehost SoC3 to execute the instruction which guarantees the order ofprocessing. Hence, as compared with the comparative example, the commandcan be acquired with low latency.

While a certain embodiment has been described, the embodiment has beenpresented by way of example only, and is not intended to limit the scopeof the inventions. Indeed, the novel embodiment described herein may beembodied in a variety of other forms; furthermore, various omissions,substitutions and changes in the form of the embodiment described hereinmay be made without departing from the spirit of the inventions. Theaccompanying claims and their equivalents are intended to cover suchforms or modifications as would fall within the scope and spirit of theinventions.

What is claimed is:
 1. A memory system comprising: a controllerincluding an interface connectable with a host with cache coherencykept, the controller being configured to: before the host writes acommand to an I/O submission queue, read the I/O submission queue; afterthe reading, detect via the interface an invalidation request, theinvalidation request being based on writing of the command by the hostto the I/O submission queue; and in response to the invalidationrequest, acquire the command in the I/O submission queue.
 2. The memorysystem according to claim 1, wherein after setting the I/O submissionqueue in initial setting, the controller is configured to perform thereading.
 3. The memory system according to claim 1, wherein in responseto the invalidation request, the controller is configured to update atail pointer of the I/O submission queue.
 4. The memory system accordingto claim 1, wherein the controller is configured to discard the readdata.
 5. The memory system according to claim 1, wherein the interfaceis in conformity with a CXL standard, and the controller is configuredto transmit a read request of the CXL standard to the host and read theI/O submission queue.
 6. The memory system according to claim 5, whereinthe controller is configured to detect the invalidation request of theCXL standard.
 7. The memory system according to claim 2, wherein theinitial setting is setting processing as to initialization processing.8. The memory system according to claim 7, wherein the initializationprocessing is performed upon turning on a power source of the memorysystem.
 9. The memory system according to claim 5, wherein the readrequest is a request to read data having a command length and a datalength of a cache management unit.
 10. The memory system according toclaim 9, wherein the controller is configured to transmit the readrequest multiple times.
 11. A method of controlling a memory systemconnectable with a host, the method comprising: connecting to the hostwith cache coherency kept; before the host writes a command to an I/Osubmission queue, reading the I/O submission queue; after the reading,detecting an invalidation request, the invalidation request being basedon writing of the command by the host to the I/O submission queue; andin response to the invalidation request, acquiring the command writtento the I/O submission queue.
 12. The method according to claim 11,wherein after setting the I/O submission queue in initial setting, thereading is performed.
 13. The method according to claim 11, wherein inresponse to the invalidation request, the method further comprisesupdating a tail pointer of the I/O submission queue.
 14. The methodaccording to claim 11, wherein the method further comprises discardingthe read data.
 15. The method according to claim 11, wherein theconnecting to the host is in conformity with a CXL standard, and themethod further comprises transmitting a read request of the CXL standardto the host and reading the I/O submission queue.
 16. The methodaccording to claim 15, wherein the detecting includes detecting theinvalidation request of the CXL standard.
 17. The method according toclaim 12, wherein the initial setting is setting processing as toinitialization processing.
 18. The method according to claim 17, whereinthe initialization processing is performed upon turning on a powersource of the memory system.
 19. The method according to claim 15,wherein the read request is a request to read data having a commandlength and a data length of a cache management unit.
 20. The methodaccording to claim 19, wherein the transmitting includes transmittingthe read request multiple times.